Assistance for Processing Pictures in Concatenated Video Streams

ABSTRACT

Providing a video stream in a transport stream, the first video stream providing a plurality of compressed pictures and information, the information including plural data fields including: a first data field corresponding to a location in the video stream of a potential splice point, the value of the first data field identifies a location in the video stream after the location of the provided information; a second data field corresponding to decompressed pictures yet to be output (DPYTBO) by a video decoder at the identified potential splice point (IPSP) when the video decoder decompresses the video stream, the value of the second data field is a number corresponding to the DPYTBO; and a third data field corresponding to pictures with contiguous output times (WCOT), the value of the third field corresponds to a set of pictures WCOT of the DPYTBO.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.12/351,776, filed Jan. 9, 2009, which claims priority to U.S.Provisional Application entitled, “SYSTEM AND METHODS FOR SPLICING ANDCONCATENATING VIDEO BITSTREAMS,” having U.S. Application Ser. No.61/019,964, filed Jan. 9, 2008, now expired, both of which are entirelyincorporated herein by reference.

This application is related to copending U.S. utility applicationentitled, “INDICATING PICTURE USEFULNESS FOR PLAYBACK OPTIMIZATION,”having U.S. application Ser. No. 11/831,916, filed Jul. 31, 2007, nowU.S. Pat. No. 8,875,199, issued Oct. 28, 2014, which claims priority toU.S. Provisional Application, “SYSTEM AND METHOD FOR SIGNALINGCHARACTERISTICS OF PICTURES' INTERDEPENDENCIES,” having U.S. ApplicationSer. No. 60/865,644, filed on Nov. 13, 2006, now expired, both of whichare entirely incorporated herein by reference.

This application is related to copending U.S. utility applicationentitled, “CONVEYANCE OF CONCATENATION PROPERTIES AND PICTURE ORDERNESSIN A VIDEO STREAM,” having U.S. application Ser. No. 12/252,632, filedOct. 16, 2008, which claims priority to U.S. Provisional Applicationentitled, “SPLICING AND PROCESSING VIDEO AND OTHER FEATURES FOR LOWDELAY,” having U.S. Application No. 60/980,442, filed Oct. 16, 2007, nowexpired, both of which are entirely incorporated herein by reference.

TECHNICAL FIELD

Particular embodiments are generally related to processing of videostreams.

BACKGROUND

Broadcast and On-Demand delivery of digital audiovisual content hasbecome increasingly popular in cable and satellite television networks(generally, subscriber television networks). Various specifications andstandards have been developed for communication of audiovisual content,including the MPEG-2 video coding standard and AVC video codingstandard. One feature pertaining to the provision of programming insubscriber television systems requires the ability to concatenate videosegments or video sequences, for example, as when inserting televisioncommercials or advertisements. For instance, for local advertisements tobe provided in national content, such as ABC news, etc., suchprogramming may be received at a headend (e.g., via a satellite feed),with locations in the programming allocated for insertion at the headend(e.g., headend encoder) of local advertisements. Splicing technologythat addresses the complexities of AVC coding standards is desired.

BRIEF DESCRIPTION OF THE DRAWINGS

Many aspects of the disclosure can be better understood with referenceto the following drawings. The components in the drawings are notnecessarily to scale, emphasis instead being placed upon clearlyillustrating the principles of the disclosed embodiments. Moreover, inthe drawings, like reference numerals designate corresponding partsthroughout the several views.

FIG. 1 is a block diagram that illustrates an example environment inwhich an embodiment of the disclosed apparatuses and methods may beemployed.

FIG. 2 is a block diagram of an example video processing device,embodied as a digital home communication terminal (DHCT), which includesfunctionality to receive, decode, and output a compressed video streamcomprising concatenated video sequences according to informationincluded in the video stream.

FIG. 3 is a block diagram that illustrates an example video streamcomprising information that assists splice operations and/or pictureoutput behavior.

FIG. 4 is a block diagram that illustrates a concept of consecutivepicture output order in a decoded picture buffer as related to theinformation conveyed in a video stream.

FIG. 5 is a flow diagram that illustrates an example method embodimentthat conveys information in a video stream from an encoder to one ormore video processing devices.

FIG. 6 is a flow diagram that illustrates another example methodembodiment that conveys information in a video stream from an encoder toone or more video processing devices.

FIG. 7 is a flow diagram that illustrates another example methodembodiment that conveys information in a video stream from an encoder toone or more video processing devices.

FIG. 8 is a flow diagram that illustrates an example method embodimentthat receives information in a video stream at a video processingdevice.

FIG. 9 is a flow diagram that illustrates another example methodembodiment that receives information in a video stream at a videoprocessing device.

FIG. 10 is a flow diagram that illustrates another example methodembodiment that receives information in a video stream at a videoprocessing device.

DESCRIPTION OF EXAMPLE EMBODIMENTS Overview

In one embodiment, a method that provides a video stream in a transportstream, the first video stream comprising a plurality of compressedpictures, and that provides information in the video stream, wherein theinformation includes plural data fields, the provided informationcomprising: a first data field corresponding to a location in the videostream of a potential splice point in the video stream, wherein thevalue of the first data field identifies a location in the video streamafter the location of the provided information in the video stream; asecond data field corresponding to decompressed pictures yet to beoutput (DPYTBO) by a video decoder at the identified potential splicepoint (IPSP) when the video decoder decompresses the video stream,wherein the value of the second data field is a number corresponding tothe DPYTBO by the video decoder at the IPSP; and a third data fieldcorresponding to pictures with contiguous output times (WCOT), whereinthe value of the third field corresponds.

Example Embodiments

Certain system, apparatus, and method embodiments are disclosed thatprovide (and/or receive) information in a video stream to assist in theimplementation of video processing operations at various devicesthroughout a subscriber television network. In one embodiment,information is conveyed by an encoding device (e.g., encoder) to a videoprocessing device embodied as a splicer (or splicing device) to indicatewhere among subsequently transmitted compressed pictures one or morepotential (e.g., suitable) splicing points occur. The information may beconveyed in some embodiments as auxiliary information or as a message,such as an SEI (supplemental enhanced information) message, in the videostream (e.g., not at a transport layer). Alternatively, or in addition,such information may include an indication of the machine state of adecoded picture buffer (DPB) resident in a second video processingdevice (e.g., digital home communication terminal or DHCT), the machinestate corresponding to the potential splicing point, which enables thesplicer to concatenate, or in some embodiments, replace a video sequenceprovided by the encoding device with a second video sequence (e.g.,commercial, such as provided by a local feed) during a transitionperiod. The information may also include control information, whichenables the DHCT to provide a seamless output from the DPB over atransition period, as well as (or alternatively) cues to variouslocations in the video stream. Additional description of these and otherembodiments are disclosed below.

A description of the MPEG-2 Video Coding standard can be found in thefollowing publication, which is hereby incorporated by reference: (1)ISO/IEC 13818-2, (2000), “Information Technology—Generic coding ofmoving pictures and associated audio—Video.” A description of the AVCvideo coding standard can be found in the following publication, whichis hereby entirely incorporated by reference: (2) ITU-T Rec. H.264(2005), “Advanced video coding for generic audiovisual services.” Adescription of MPEG-2 Systems for transporting AVC video streams inMPEG-2 Transport packets can be found in the following publications,which are hereby entirely incorporated by reference: (3) ISO/IEC13818-1, (2000), “Information Technology—Generic coding of movingpictures and associated audio—Part 1: Systems,” and (4) ITU-T Rec.H.222.0|ISO/IEC 13818-1:2000/AMD.3, (2004), “Transport of AVC video dataover ITU-T Rec. H222.0|ISO/IEC 13818-1 streams.” Additionally, alsopublished is ITU Rec H.264/ISO/IEC 14496 Part 10, which is incorporatedherein by reference.

FIG. 1 is a high-level block diagram depicting an example environment inwhich an embodiment of apparatuses, systems, and/or methods that conveyand use information corresponding to potential splice points and decodedpicture buffer state information (e.g., machine state, such asconsecutive picture order, etc.) are implemented. In particular, FIG. 1is a block diagram that depicts an example subscriber television system(STS) 100. In this example, the STS 100 includes a headend 110 and oneor more video processing devices, the latter embodied in this example asa DHCT 200. The DHCT 200 and the headend 110 are coupled via a network130. The headend 110 and the DHCT 200 cooperate to provide a user withtelevision services including, for example, broadcast video programs, aninteractive program guide (IPG), and/or video-on-demand (VOD)presentations, among others. The television services are presented viathe display device 140, which is typically a television set that,according to its type, is driven with an interlaced scan video signal ora progressive scan video signal. However, the display device 140 mayalso be any other device capable of displaying video images including,for example, a computer monitor, a mobile phone, game device, etc.Although shown communicating with a display device 140, the DHCT 200 maycommunicate with other devices that receive, store, and/or process videostreams from the DHCT 200, or that provide or transmit video streams oruncompressed video signals to the DHCT 200.

The DHCT 200 is typically situated at a user's residence or place ofbusiness and may be a stand-alone unit or integrated into another devicesuch as, for example, the display device 140, a personal computer,personal digital assistant (PDA), mobile phone, among other devices. TheDHCT 200 receives signals (video, audio and/or other data) including,for example, digital video signals in a compressed representation of adigitized video signal such as, for example, AVC streams modulated on acarrier signal, and/or analog information modulated on a carrier signal,among others, from the headend 110 through the network 130, and providesreverse information to the headend 110 through the network 130. The DHCT200 comprises, among other components, a decoded picture buffer (DPB),as explained further below.

The network 130 may include any suitable medium for communicating videoand television service data including, for example, a cable televisionnetwork or a satellite television network, among others. The headend 110may include one or more server devices (not shown) for providing video,audio, and other types of media or data to client devices such as, forexample, the DHCT 200.

The headend 110 also includes one or more encoders (encoding devices,compression engines) 111 (one shown) and one or more video processingdevices embodied as one or more splicers 112 (one shown) coupled to theencoder 111. In some embodiments, the encoder 111 and splicer 112 may beco-located in the same device and/or in the same locale (e.g., both inthe headend 110 or elsewhere), while in some embodiments, the encoder111 and splicer 112 may be distributed at different locations within theSTS 100. The systems and methods disclosed herein are applicable to anyvideo compression method performed according to a video compressionspecification allowing for at least one type of compressed picture thatcan depend on the corresponding decompressed version of each of morethan one reference picture for its decompression and reconstruction. Forexample, the encoder 111 may compress the inputted video signal (e.g.,from a provider or from a headend server) according to the specificationof the AVC standard and produce an AVC stream containing different typesof compressed pictures, some that may have a first compressed portionthat depends on a first reference picture for their decompression andreconstruction, and a second compressed portion of the same picture thatdepends on a second and different reference picture. The compressedvideo (and audio) streams are produced in accordance with the syntax andsemantics of a designated video (and audio) coding method, such as, forexample, MPEG-2 or AVC, so that the compressed video (and audio) streamscan be interpreted by a decompression engine 222 (FIG. 2) fordecompression and reconstruction at a future time. Each AVC stream ispacketized into transport packets according to the syntax and semanticsof transport specification, such as, for example, MPEG-2 transportdefined in MPEG-2 systems. Each transport packet contains a header witha unique packet identification code, or PID, associated with therespective AVC stream.

In one embodiment, the encoder 111 provides a compressed video stream ina transport stream with information that conveys to the splicer 112 oneor more suitable splice points as described further below. The splicer112 splices one or more video sequences of a second video stream (e.g.,provided by a video source separate than the video source that providesthe first video stream) to the video stream provided by the encoder 111,and/or in some embodiments, replaces one or more of the video sequencesprovided by the encoder 111 with other video sequences. Further, thesplicer 112 may pass the information provided by the encoder 111, withor without modification, to the DHCT 200, and/or generate additionalcontrol information based on the received information, the controlinformation received at and used by the DHCT 200 to enable a seamlessoutput of the pictures (decoded pictures) from the DPB of the DHCT 200across a concatenation.

Any of the below described subsystems or methods of DHCT 200, encoder111, and/or splicer 112 can comprise an ordered listing of executableinstructions for implementing logical functions, can be embodied in anycomputer-readable medium for use by or in connection with an instructionexecution system, apparatus, or device, such as a computer-based system,processor-containing system, or other system that can fetch theinstructions from the instruction execution system, apparatus, or deviceand execute the instructions.

FIG. 2 is a block diagram of an embodiment of a DHCT 200 as depicted inFIG. 1, in which an embodiment of systems and methods that implement, atleast in part, processing of compressed video and information used inenabling the output of pictures from the DPB. It will be understood thatthe DHCT 200 shown in FIG. 2 is merely illustrative and should not beconstrued as implying any limitations upon the scope of the disclosure.For example, in some embodiments, the DHCT 200 may have fewer,additional, and/or different components than the components illustratedin FIG. 2.

The DHCT 200 preferably includes a communications interface 242 forreceiving signals (video, audio and/or other data) from the headend 110(FIG. 1) through the network 130 (FIG. 1), and provides reverseinformation to the headend 110.

The DHCT 200 may further include one or more processors (one processor244 is shown) for controlling operations of the DHCT 200, an outputsystem 248 for driving the television display 140 (FIG. 1), and a tunersystem 245 for tuning to a particular television channel and/orfrequency and for sending and receiving various types of data to/fromthe headend 110 (FIG. 1). The DHCT 200 may include, in some embodiments,multiple tuners for receiving downloaded (or transmitted) data. Thetuner system 245 can select from a plurality of transmission signalsprovided by the subscriber television system 100 (FIG. 1). The tunersystem 245 enables the DHCT 200 to tune to downstream media and datatransmissions, thereby allowing a user to receive digital media contentvia the subscriber television system 100. In one embodiment, analog TVsignals can be received via tuner system 245. The tuner system 245includes, in one implementation, an out-of-band tuner for bi-directionaldata communication and one or more tuners (in-band) for receivingtelevision signals. Additionally, a receiver 246 receivesexternally-generated user inputs or commands from an input device suchas, for example, a remote control device (not shown).

The DHCT 200 may include one or more wireless or wired interfaces, alsocalled communication ports or interfaces 274, for receiving and/ortransmitting data or video streams to other devices. For instance, theDHCT 200 may feature USB (Universal Serial Bus), Ethernet, IEEE-1394,serial, and/or parallel ports, etc. The DHCT 200 may be connected to ahome network or local network via communication interface 274. The DHCT200 may also include an analog video input port for receiving analogvideo signals. User input may be provided via an input device such as,for example, a hand-held remote control device or a keyboard.

The DHCT 200 includes at least one storage device 273 for storing videostreams received by the DHCT 200. A PVR application 277, in cooperationwith operating system 253 and device driver 211, effects among otherfunctions, read and/or write operations to/from the storage device 273.The processor 244 may provide and/or assist in control and programexecution for operating system 253, device driver 211, applications(e.g., PVR 277), and data input and output. The processor 244 mayfurther provide and/or assist in receiving and processing informationcorresponding to splice points and/or machine state in the receivedvideo stream, and decoding and outputting a video stream according tothe received information. Herein, references to write and/or readoperations to the storage device 273 can be understood to includeoperations to the medium or media of the storage device 273. The devicedriver 211 is generally a software module interfaced with and/orresiding in the operating system 253. The device driver 211, undermanagement of the operating system 253, communicates with the storagedevice controller 279 to provide the operating instructions for thestorage device 273. As conventional device drivers and devicecontrollers are well known to those of ordinary skill in the art,further discussion of the detailed working of each will not be describedfurther here.

The storage device 273 may be located internal to the DHCT 200 andcoupled to a common bus 205 through a communication interface 275. Thecommunication interface 275 may include an integrated drive electronics(IDE), small computer system interface (SCSI), IEEE-1394 or universalserial bus (USB), among others. Alternatively or additionally, thestorage device 273 may be externally connected to the DHCT 200 via acommunication port 274. The communication port 274 may be according tothe specification, for example, of IEEE-1394, USB, SCSI, or IDE. In oneimplementation, video streams are received in the DHCT 200 viacommunications interface 242 and stored in a temporary memory cache (notshown). The temporary memory cache may be a designated section of DRAM252 or an independent memory attached directly, or as part of acomponent in the DHCT 200. The temporary cache is implemented andmanaged to enable media content transfers to the storage device 273. Insome implementations, the fast access time and high data transfer ratecharacteristics of the storage device 273 enable media content to beread from the temporary cache and written to the storage device 273 in asufficiently fast manner. Multiple simultaneous data transfer operationsmay be implemented so that while data is being transferred from thetemporary cache to the storage device 273, additional data may bereceived and stored in the temporary cache.

The DHCT 200 includes a signal processing system 214, which comprises ademodulating system 210 and a transport demultiplexing and parsingsystem 215 (herein demultiplexing system) for processing broadcastand/or on-demand media content and/or data. One or more of thecomponents of the signal processing system 214 can be implemented withsoftware, a combination of software and hardware, or in hardware. Thedemodulating system 210 comprises functionality for demodulating analogor digital transmission signals.

An encoder or compression engine, as explained above, may reside at theheadend 110 (e.g., embodied as encoder 111), in the DHCT 200 (e.g.,embodied as compression engine 217), or elsewhere. The compressionengine 217 can receive a digitized uncompressed video signal, such as,for example, one provided by analog video decoder 216, or a decompressedvideo signal produced by a decompression engine (e.g., decompressionengine 222) as a result of decompressing a compressed video signal.

In one embodiment, digitized pictures and respective audio output by theanalog video decoder 216 are presented at the input of the compressionengine 217, which compresses the uncompressed sequence of digitizedpictures according to the syntax and semantics of a video compressionspecification. Thus, the compression engine 217 implements a videocompression method or algorithm that corresponds to a respective videocompression specification, such as the AVC standard, to provide a videostream.

In some embodiments, a compression engine with similar compressioncapabilities, such as one that can produce AVC streams, is connected tothe DHCT 200 via communication port 274, for example, as part of a homenetwork. In another embodiment, a compression engine with similarcompression capabilities, such as one that can produce AVC streams, maybe located at the headend 110 or elsewhere in the network 130, asexplained above. In some embodiments, the compression engine 217 in mayinclude capabilities to provide splicing information (e.g., includingcontrol information) for a produced video stream, such as when locallystored video sequences (e.g., commercials stored in the storage device273) are intended for local concatenation (e.g., locally spliced) andsubsequent decoding and output.

The demultiplexing system 215 can include MPEG-2 transportdemultiplexing capabilities. When tuned to carrier frequencies carryinga digital transmission signal, the demultiplexing system 215 enables theseparation of packets of data, corresponding to the desired AVC stream,for further processing. Concurrently, the demultiplexing system 215precludes further processing of packets in the multiplexed transportstream that are irrelevant or not desired, such as packets of datacorresponding to other video streams. Parsing capabilities of thedemultiplexing system 215 allow for the ingesting by the DHCT 200 ofprogram associated information carried in the transport packets.

The components of the signal processing system 214 are generally capableof QAM demodulation, forward error correction, demultiplexing of MPEG-2transport streams, and parsing of packets and streams. Stream parsingmay include parsing of packetized elementary streams or elementarystreams. In yet another embodiment, processor 244 performs parsing,processing, and interpretation of information. The signal processingsystem 214 further communicates with the processor 244 via interrupt andmessaging capabilities of the DHCT 200. The processor 244 annotates thelocation of pictures within the video stream or transport stream as wellas other pertinent information corresponding to the video stream.

The packetized compressed streams can also be outputted by the signalprocessing system 214 and presented as input to the decompression engine222 for audio and/or video decompression. The signal processing system214 may include other components (not shown), including memory,decryptors, samplers, digitizers (e.g., analog-to-digital converters),and multiplexers, among others.

Upon effecting the demultiplexing and parsing of the transport stream,the processor 244 interprets the data output by the signal processingsystem 214 and generates ancillary data in the form of a table or datastructure (index table 202) comprising the relative or absolute locationof the beginning of certain pictures in the compressed video stream.

The demultiplexing system 215 can parse the received transport stream(or the stream generated by the compression engine 217, which in someembodiments may be a program stream) without disturbing its video streamcontent and deposit the parsed transport stream (or generated programstream) into the DRAM 252. The processor 244 causes the transport streamin DRAM 252 to be transferred to a storage device 273. Additionalrelevant security, authorization and/or encryption information may bestored.

A decompression engine 222 comprises one or more software and/orhardware components (collectively logic) including video decompressionlogic 223, upconverter logic 227, audio decompression logic 225, and aDENC 226. Note that in some embodiments, one or more of theabove-described logic may be distributed among several components, oromitted in some embodiments. The decompression engine 222 providesdecoding functionality responsive in some embodiments to informationreceived in the stream, the information identifying splice points orindicators of splice points or potential splice points, and/or outputbehavior of one or more pictures of the decoded picture buffer (DPB) 297residing in one embodiment in decompression engine memory 299. Note thatat least in one embodiment, reference herein to a decoding systemcomprises decoding functionality and cooperating elements, such as foundin the collective functionality of the decompression engine 222,processor 244, signal processing system 214, and memory. In someembodiments, the decoding system can comprise fewer, greater, ordifferent elements. Further, certain system and method embodimentsinclude components from the headend (e.g., the encoder 111, etc.) and/orcomponents from the DHCT 200, although a fewer or greater amount ofcomponents may be found in some embodiments.

Having provided example environments in which certain embodiments of thedisclosed systems and methods may be employed, attention is directed toFIGS. 3 and 4 in conjunction with the following description. It is notedthat some of the below description pertaining to the conveyance ofinformation (e.g., conveyance of N and M, as described below) may alsobe found, at least in part, in one or more of the commonly assigned andco-pending applications cited in the cross-reference section above. Withregard to FIG. 3, shown in an example video stream 300, for instance asprovided by the encoder 111. In some embodiments, the video stream 300may be provided in a transport stream. A message 302 provided in thevideo stream 300 includes information that conveys the relative locationfrom a current picture (picture, frame, or access unit) or the locationof the message 302 in the bitstream 300 to, for example, immediatelyafter a second access unit (the second access unit designated forillustrative purposes as N₀, and the location immediately after thesecond access unit in the bitstream 300 is designated as 304) toidentify a location or a property. The number of pictures or accessunits, N, identifies the location in the bitstream 300 (i.e., the videostream), and N may be expressed in the message 302 in one of differentforms that connotes the number of consecutive frames, access units,pictures, slices, fields, or time ticks, or in the bitstream to identifya location (e.g., 304) from the location of the provided message 302.Different embodiments express N differently, as shown below. One methoddoes not convey N in the message 302.

Thus, N expresses the number of consecutive pictures or access unitsbetween the two locations (e.g., 302 and 304) in the bit stream 300. Adata field is a data element that conveys the value for N. The dataelement may be pre-assigned to a particular “non-zero” value that issufficient to identify the N^(th) picture in the bit stream 300 (orherein AVC stream) as the picture or point in the bitstream 300identifying particular property or location. Note that in someembodiments, the message 302 may provide a cue to additional potentialsplice points, such as potential splice point 306 immediately followingthe N_(I) ^(th) picture and immediately preceding the N_(I)+1^(th)picture in the video stream 300.

Information conveying a stream property provides: (1) a location in theAVC stream where the stream property becomes effective, and (2)information related to, and identifying, a particular stream property.The conveyed information enables a video processing device to perform acorresponding stream manipulation operation that is suitable to beperformed at the location in the AVC stream where the identifiedparticular stream property is effective. For instance, a first streamproperty may be a “suitable splice point,” which is a location in theAVC stream suitable for transitioning from the AVC stream into anotherAVC stream, such as an advert or commercial. A suitable splice point maybe also called a potential splice point or an identified or determinedsplice point.

In one embodiment, the message 302 includes a form of providing a valuefor N consecutive pictures in a first AVC stream (e.g., 300)corresponding to the location for a “suitable splice point” (e.g., 304)that allows a stream splicing operation to performed by a splicingdevice from the first bitstream 300 to a second bitstream (not shown). Afirst data element provides a value, N, corresponding to the number ofconsecutive pictures, including the first picture, to identify thelocation 304 after the N^(th) picture and prior to the (N+1)^(th)picture as the location in the first AVC stream 300 where the “suitablesplice point” becomes effective. A second data element provides a value,M, that conveys the number of decompressed pictures in the decodedpicture buffer (DPB) (e.g., DPB 297, FIG. 2) that have successivepicture-output times, with the first of the successive output timesbeing at the picture-output time immediately after the decompressiontime of the N^(th) picture. The DPB 297 is in accordance with the AVC(also ITU H.264) standard and resides in decompression memory (e.g.,decompression engine memory 299) of a decoder (e.g., decompressionengine 222). The location 304 in the AVC stream 300 where the “suitablesplice point” becomes effective equals the decompression time of theN^(th) picture (i.e., under the assumption of a hypotheticalinstantaneous decoder). The earliest output time of the M decompressedpictures of the first AVC stream residing in the DPB,output-time(1_of_M), equals the decompression time of the N^(th) pictureplus one picture-output interval. That is, it is at the nextpicture-output time, thus output(1_of_M)=decode-time(N_of_N)+1.

The M decompressed pictures in the DPB with successive output times mayhave been in successive order in the first AVC stream 300. In oneembodiment, the corresponding compressed M pictures were not insuccessive order in the first AVC stream 300.

The number of picture-output times from the decompression of the firstof the N consecutive picture, decode-time (1_of_N), to thepicture-output time of the last of the M pictures in the DPB,output-time(M_of_M), equals (N+M). Hence, there are (N+M) differentpictures that are output from the first AVC stream up to the “suitablesplice point.” Each of the (N+M) different pictures has a respectiveoutput-time corresponding to one of (N+M) consecutive picture-outputtimes, the first picture-output time being coincident withdecode-time(1_of_N).

In one embodiment, a first AVC stream is required to exhibit thefollowing properties at the location where the “suitable splice point”becomes effective:

-   -   1. An AVC decompression engine 222 (FIG. 2) that receives and        decompresses a portion of the first AVC stream, that ends with        the N^(th) picture and includes the N consecutive pictures, must        be able to:        -   A. Output all of the N consecutive pictures between the            (N+M) picture-output times starting with and including            decode-time(1_of_N and ending with output-time(M_of_M), and        -   B. Output (N+M) different pictures during these (N+M)            consecutive picture-output times.    -   2. No picture in the first AVC stream prior to and including the        N^(th) picture must have an output time after        output-time(M_of_M).

In summary, in one embodiment, a compression engine (e.g., encoder 111)or video processing device (e.g., splicer 112) may not provide a“suitable splice point” that results in a discontinuity or gap for anyof the (M+N) picture-output times, possibly forcing a previously outputpicture to be output repeatedly (i.e., because the picture correspondingto a respective picture-output time was not in the first AVC streamprior to the “suitable splice point”). If a picture had an output timeafter output-time(M_of_M), it would reside in the DPB 297 and not beoutput. A video processing device and/or compression engine provides amessage corresponding to a “suitable splice point” only if thecorresponding location in the stream satisfies the above properties.

A splice operation of the first AVC stream to the second AVC stream isperformed by a video splicing device (e.g., splicer 112, FIG. 2),located in the headend 110, network 130, or elsewhere, by using valuesof the first data element and second data element, N and M,respectively, provided in a message (e.g., message or information 302)in the bitstream 300 (i.e., the compressed video stream or AVC stream).The video splicing device 112 uses the N and M values to produce a thirdAVC stream comprising of a portion of the first AVC stream followed bythe second AVC stream. The portion of the first AVC stream in the thirdAVC stream terminates at the location of the first AVC stream after theN^(th) picture. The first picture of the second AVC stream (FPOSAS) thatfollows the N^(th) picture in the third stream is referred to as theFPOSAS-picture. The video splicing device 112 produces the third AVCstream with an overlapped transition period of M picture-output times.That is, the video splicing device 112 produces the third AVC streamsuch that following four conditions are met:

-   -   1. the M pictures from the first AVC stream with successive        output times and residing in the DPB buffer at a “suitable        splice point,” are assigned a respective output (or display)        time for each of the M picture-output times in accordance with        their original output order,    -   2. None of the pictures from the first AVC stream are        decompressed during the overlapped transition period. That is,        the latest decode time assigned to a picture from the first AVC        stream is prior to the start of the overlapped transition        period.    -   3. M pictures from the second AVC stream, starting with the        FPOSAS-picture, are decompressed during the overlapped        transition period.    -   4. None of the pictures from the second AVC stream are output        during the overlapped transition period. The earliest        picture-output time assigned to a picture from the second AVC        stream is one picture-output time after the end of the        overlapped transition period.

Referring to FIG. 4, shown is a block diagram that illustrates,generally, picture interdependencies in an example video sequence ofcompressed pictures provided by the encoder 111, and more specifically,the conditions described above. The first row 402 of FIG. 4 comprisesthe output order of an exemplary GOP, such as received and decoded indecode order 404 (i.e., transmission order) at the decompression engine222. The GOP comprises a sequence of compressed pictures (designated bypicture types, such as I, P, or B). The picture interdependencies areshown, in part, by the arrows above and below each picture (picturesnumbered 1-25 at the top of each respective picture). An arrow tailshown at a picture depicts that that picture serves as a referencepicture to the other picture where the corresponding arrow head isshown. That is, the arrow conveys that the other picture is predictedfrom the reference picture. For instance, in FIG. 4, picture 3 ispredicted from (and hence depends from) picture 1 and picture 5.Although shown described having a similar pattern of dependency betweenbounds (e.g., between pictures 1 and 9, and pictures 9 and 17), onehaving ordinary skill in the art should appreciate that differentpatterns and/or dependencies can be found from one bounded group to thenext.

Beneath row 402 of FIG. 4 is the decode order 404, corresponding to theorder in which the pictures are decoded for reconstruction at thedecompression engine 222. The transmission order of pictures isdifferent than the output or display order due to the need to have thereference pictures prior to decoding a picture. Note that P pictures canbe forward predicted or backwards predicted, and typically, that fact isnot evident until the pictures are decoded. For instance, knowledge ofthe picture type (e.g., as ascertained by a header) does not necessarilyconvey how prediction is employed or picture interdependencies.

Beneath row 404 is section 406, which illustrates the machine state ofthe DPB 297, sorted (from top to bottom in 406) in order of output fromthe DPB 297. Below section 406 is row 408, which provides a value for M.For instance, it is noted that potential splice points 410 and 412 eachfollow a machine state of the DPB 297 that comprises three (M=3)pictures in consecutive output order (e.g., B₇, b₈, P₉ for potentialsplice point 410 and B₁₅, b₁₆, and P₁₇ for potential splice point 412).In some embodiments, other values of M correspond to potential splicepoints.

The third AVC stream is received by DHCT 200 and decompression isperformed on the compressed picture of the third AVC stream bydecompression engine 222. DHCT 200 is an example of a receiver thathouses a decoder (i.e., a decoding device, such as decompression engine222). Decompressed pictures are stored in the decompression memory 299.The output system 248 serves to output (e.g., to display device 140) thedecompressed pictures at their respective output times. Orchestration ofdecompression and outputting of pictures is performed according to therespective decode-time and output-time of each picture in the third AVCstream. When the portion of the third AVC stream corresponding to thespliced first and second AVC streams is processed by decompressionengine 222, the overlapped transition period comes into effect. Duringthe overlapped transition period, decompression engine 222 decompressesM consecutive pictures that emanated from the second AVC stream whileoutputting the last M pictures from the first AVC stream.

In one embodiment, at least one of the N consecutive pictures prior tothe identified “suitable splice point” in the first AVC stream is alsoone of the M decompressed pictures of the first AVC stream in the DPB297 with successive output times at the time that the “suitable splicepoint” becomes effective.

In one embodiment, N is required to be greater than M to announce the“suitable splice point” in the first AVC stream with sufficient leadtime before it becomes effective. In an alternate embodiment, N>M and Nis also greater than a pre-specified threshold (e.g., threepicture-output intervals or picture-output times). In yet anotherembodiment, the same “suitable splice point” is announced N times withrespective messages corresponding respectively to each of the Nconsecutive picture in the first AVC stream. That is, starting with thefirst of the N consecutive in the first AVC stream, N instances ofmessages is provided in the video stream, each instance correspondingrespectively to one of the N consecutive pictures. The information inthe message conveys respective values for the first data element andsecond data element as necessary to provide an indication of a potentialor suitable splice point.” The first data element's value is N for thefirst picture and decreases by one successively in each successiveinstance of the message and corresponds to each one of the successiveaccess units in the sequence of N consecutive access units. The firstdata element's value, N, finally becomes equal to one for the N^(th)picture. The second data element's value remains constant, equal to M,through the N successive instances of the message that respectivelycorresponds to the N consecutive pictures. Two “different devices” mayuse two different announced messages that convey a potential splicepoint in the first AVC stream to prepare and perform the transition tothe second AVC stream at the identified location of the “suitable splicepoint” of the first AVC stream. A third “video splicing device” may usemore than one, and possibly all N instances of messages to prepare andperform the transition.

In one embodiment, the video splicing device that produces the third AVCstream sets the decompression time for the FPOSAS-picture equal tooutput-time(1_of_M), which is also equal to the decode-time(N_of_N)+1.

In one embodiment, the video splicing device 112 provides theFPOSAS-picture with an output time equal to M picture-output times afterits decompression time and the FPOSAS-picture serves as a past referencepicture to at least one picture with a decode-time greater than thedecode-time of the FPOSAS-picture. Thus, M picture-output times areadded to the decode-time of the FPOSAS-picture. In another embodiment,the FPOSAS-picture in the third AVC stream is provided a picture-outputtime less than M picture-output times after its decompression time,forcing a shortened overlapped transition period and at least one of theM pictures from the portion of the first AVC stream to not be displayed.In yet another embodiment, the output time of the FPOSAS-picture isgreater than or equal to (M+1) picture-output times after itsdecode-time, and the FPOSAS-picture serves as a future reference pictureto at least one picture with a decode-time greater than the decode-timeof the FPOSAS-picture, including the picture from the second AVC streamthat has an output time equal M picture-output times after thedecompression time of the FPOSAS-picture.

The FPOSAS-picture in the third AVC may be an IDR-picture. In anotherembodiment, the FPOSAS-picture in the third AVC may be an IDR-picture oran I-picture. In yet another embodiment, the FPOSAS-picture in the thirdAVC stream is an I-picture.

In one embodiment, a compression engine 111 that produces the first AVCstream provides each picture in the first AVC stream with theirrespective picture-output time delayed by one picture-output interval tocause the value of M to be increased by one. Although the maximum numberof reference pictures that can be retained in the DPB 297 is reduced byone, it benefits the splicing operation by lengthening the overlappedtransition period from the first AVC stream to the second AVC stream byone picture-output interval. The longer overlapped transition periodtends to reduce any potential increase in the bit-rate of the third AVCstream that may manifest as a result of starting compression at theFPOSAS-picture without the benefit of reference pictures.

In an alternate embodiment, the video splicing device 112 provides alonger overlapped transition period by causing the last picture outputfrom the portion of the first AVC stream to be output repeatedly overone or more extra picture-output intervals and setting the respectivepicture-output times for the pictures from the portion of the second AVCstream accordingly.

In one embodiment, the video splicing device 112 producing the third AVCstream retains in the bitstream the original information that conveyedthe “suitable splice point” for the first AVC stream. The third AVCstream may then be spliced at a later time at the location in the thirdAVC stream where the “suitable splice point” becomes effective. Thus theportion of the third AVC stream containing the first AVC stream can beretained and the portion corresponding to the second AVC stream can beoverwritten, in part or in its entirety, starting with theFPOSAS-picture. As a non-limiting example, when the second AVC streamcorresponds to a first commercial, this allows for another spliceoperation to be performed to overwrite the second AVC stream by a fourthAVC stream that corresponds to a second commercial. The stream splicingoperation from the third AVC stream to the fourth AVC stream can beperformed by a different video splicing device than the one thatproduced the third AVC video stream. The produced fifth AVC streamcomprises of the portion of the first AVC stream in the third AVC streamfollowed by the fourth AVC stream.

In one embodiment, the video splicing device producing the third AVCstream uses the additional information for the “suitable splice point,to perform and enhance the splicing of the first and second AVC streams.

In one embodiment, a message conveying a “suitable splice point” andcorresponding to the first of N consecutive pictures in the first AVCstream also includes a third data element that provides a valuecorresponding to P consecutive pictures prior to, but not including, thefirst of N consecutive pictures (i.e., the identified picture). WhereasN conveys the location in the first AVC stream where the “suitablesplice point” becomes effective, P conveys the number of consecutivepictures in the first AVC stream that must be decompressed prior to thefirst of the N consecutive pictures so that all (N+M) pictures can beoutput with their complete information. For instance, if a user hasmerely started receiving a broadcast video program, it may not bepossible to obtain all the information to decompress some pictures thatdepend on reference pictures that were transmitted prior to when theuser started receiving the program. Likewise, some pictures mayindirectly depend on some reference pictures that are not available. Inan alternate embodiment, P may be the number of pictures that must bedecompressed prior to the N^(th) picture, and P>N. In anotherembodiment, P pictures must be decompressed to guarantee the output withcomplete information of the M pictures in the DPB.

In yet another embodiment, a message conveying information for apotential (i.e., suitable) “splice point” is only provided at a locationin the AVC stream that guarantees the output with complete informationof the M pictures in the DPB. Alternatively, it is only provided at alocation in the AVC stream that guarantees the output with completeinformation of the (N+M) pictures.

Conveyance of Suitable Splice Points in the Bitstream

Having provided a general overview for the various systems and methodsdescribed herein, attention is directed to FIGS. 5-7 and FIGS. 8-10.FIGS. 5-7 are flow diagrams that illustrate example methods implementedin one embodiment by the encoder 111 to convey information (e.g.,messages) to a splicer 112, the information including a cue to potentialsplice points and/or the machine state of pictures in the DPB 297. FIGS.8-10 are flow diagrams that illustrate example methods, implemented inone embodiment by a splicer 112, that receive and process theinformation and video stream provided by the encoder 111 to assist inthe concatenation (splicing) process. In general, one or moreembodiments of the example methods provide (or receive) a messageconveying a potential splice point in the bitstream located N accessunits from the location in the bitstream of the message or the currentaccess unit. The message also conveys for the identified splice pointthe number of access units in the DPB 297 (decoded picture buffer at thedecoder), M, as explained above. A splicing device (e.g., splicer 112)may use the identified splice point to concatenate two bitstreams,imparting at the decoder 222 a splice transition period that spans Mpicture output intervals. During the splice transition period at thedecoder 222, a picture from the first bitstream is output from the DPB297 while each decoded picture from the second bitstream enters the DPB297.

The provided message may serve as a cue that would be provided by theencoder 111 to convey a suitable splice point in the bitstream to thesplicing device 112 (e.g., the receiving device). The splicing device112, in one embodiment, may implement a method of receiving the providedmessage, interpreting the information in the provided message todetermine whether to exercise initiating and splice operation at anidentified potential splice point. The splicing device 112 performs asplicing operation based on one or more received messages, eachcontaining information pointing to distinctive splice points in thebitstream. In another embodiment, several messages are received at thesplicing device 112 but one or more of the messages point to the samesplice point in the bitstream.

A splice point is located N access units from the current access unit orthe location from where the message is located in the bitstream. Herein,a potential splice point is characterized as: (1) a location in thebitstream in which M pictures in the DPB have successive picture-outputtimes, and (2) the earliest output time of the M pictures equals theoutput time immediately after the decode time of the N^(th) picture(e.g., the decode time of the Nth picture plus one picture-outputinterval).

The splicing device 112 that uses the received message (e.g., 302, FIG.3) can use the information to cause a transition period at the decoder222 in which pictures from the first bitstream in the DPB 297 are outputwhile pictures from the second bitstream are decoded and stored in theDPB.

As N identifies the location of a potential splice point in thebitstream, it may be expressed in the message in one of different formsthat connotes the number of consecutive frames, access units, or slicesin the bitstream to identify the location of a suitable splice pointfrom the location of the provided message. Different embodimentsexpressing N differently, as shown below. One method does not convey Nin the message but M only.

A potential splice point may be announced in one or more, and possiblyall of the N access units (i.e., an access unit is a distinct encodedpicture in the compressed video stream, or bitstream, that precedes thelocation of the potential splice point). The value of N is decrementedin successive instances of the message in consecutive access units inthe bitstream that precede the identified splice point

The splicing device or splicer 112 may use an identified splicing pointto produce a concatenation of two bitstreams. At the decoder 222, theconcatenated bitstream exhibits a transition period that spans Mpicture-output times. As a picture from the first bitstream is outputfrom the DPB 297 during the transition period, a corresponding picturefrom the second bitstream is decoded and stored in the DPB. A decodedpicture from the second bitstream can occupy the DPB vacancy createdfrom outputting the picture from first bitstream. The splicer 112 mayinclude MMCOs (memory management and control operations that remove adecoded picture that resides in the DPB) to remove some or all of thepictures of the first bitstream from the DPB 297. However, in someembodiments, the splicer 112 may start the second bitstream with an IDRand make proper use of no_output_of_prior_pics_flag.

The earliest picture-output time assigned to a picture from the secondbitstream by the splicer 112 is one picture-output time after lastoutput picture from the first bitstream. However, it should be notedthat in some embodiments, the splicer 112 may opt to use an identifiedsplice point but use a transition period less than M. Likewise, the DPB297 may have more than M pictures at the identified splice point and thesplicer 112 may need to employ MMCOs.

Some example message syntaxes corresponding to the conveyed informationare presented below from which various method and system embodiments arecontemplated.

Splice Point Cue1 Message Syntax

C Descriptor splice _point_cue1( payloadSize ) {N_num_fields_to_prior_pics_dpb_output 5 u(8) M_num_fields_dpb_output_delay 5 u(5) num_prior_pics_in_dpb 5 u(5) for (j = 0; j <num_prior_pics_in_dpb; j++) { consecutive_poc_flag[ i ] 5 u(1)pic_struct[ i ] 5 u(3)  }

Splice Point Cue1 Message Semantics

The splice point cue1 message is intended to provide early cues forpre-conditioned splice points. This message may provide early cues toassist digital program insertion. Concatenation devices may make use ofthis information to generate the IDR prior pictures output cue message.

N_num_fields_to_prior_pics_dpb_output specifies the time, in fieldoutput time units, from decoding of current access unit to the decodingof access unit immediately prior to the indicated splice point to whichthis message applies.

M_num_fields_(—) dpb_output_delay specifies the time, in field outputtime units, from first picture to last picture in DPB exhibitingcontiguous output times and with the first picture having output timeequal to N_num_fields_to_prior_pics_dpb_output+2. A frame is considered2 fields' time.

num_prior_pics_in_dpb indicates the number of prior pictures remainingin the DPB to be output after the decoding of the associated access unitprior to the indicated splice point. Not all pictures in the DPB mayhave consecutive output times.

consecutive_poc_flag[i] equal to 1 indicates that the picture has aconsecutive picture order count as compared to previous picture outputfrom the DPB. Picture order counts are consecutive if the increments are1 for field pictures and 2 for frame pictures, or the pictures areintended to be output without gaps. The pictures are sorted bycorresponding picture order counts from smallest value to highest value.

pic_struct[i] indicates the pic_struct value that is associated witheach picture. The pictures are sorted by corresponding picture ordercounts from smallest value to highest value.

Splice Point Cue2 Message Syntax

C Descriptor splice_point_cue2( payloadSize ) {N_num_AU_to_prior_pics_dpb_output 5 u(8) M_num_AU_dpb_output_delay 5u(5) num_prior_pics_in_dpb 5 u(5) for (j = 0; j < num_prior_pics_in_dpb;j++) { consecutive_poc_flag[ i ] 5 u(1) pic_struct[ i ] 5 u(3)  }

Splice Point Cue2 Message Semantics

The splice point cue2 message is intended to provide early cues forpre-conditioned splice points. This message may provide early cues toassist digital program insertion. Concatenation devices may make use ofthis information to generate an IDR prior pictures output cue message.

N_num_AU_to_prior_pics_dpb_output specifies the number of access units,from decoding of current access unit to the decoding of access unitimmediately prior to the indicated splice point to which this messageapplies.

M_num_AU_(—) dpb_output_delay specifies the number of access units inDPB with contiguous output times. The first picture has output timeequal to N_num_fields_to_prior_pics_dpb_output+2.

num_prior_pics_in_dpb indicates the number of prior pictures remainingin the DPB to be output after the decoding of the associated access unitprior to the indicated splice point. Not all pictures in the DPB mayhave consecutive output times.

consecutive_poc_flag[i] equal to 1 indicates that the picture has aconsecutive picture order count as compared to previous picture outputfrom the DPB. Picture order counts are consecutive if the increments are1 for field pictures and 2 for frame pictures, or the pictures areintended to be output without gaps. The pictures are sorted bycorresponding picture order counts from smallest value to highest value.

pic_struct[i] indicates the pic_struct value that is associated witheach picture. The pictures are sorted by corresponding picture ordercounts from smallest value to highest value.

Splice Point Cue3 Message Syntax

C Descriptor splice _point_cue3( payloadSize ) {N_num_AU_to_prior_pics_dpb_output u(8) M_num_AU_ dpb_output_delay 5 u(4)num_prior_pics_in_dpb u(4) last_pic_before_splice_POC  u(32) if(CpbDpbDelaysPresentFlag ) { splice_pic_initial_buffering_delay 5 u(v)splice_pic_dpb_output_delay 5 u(v) } }

Splice Point Cue3 Message Semantics

The splice point cue3 message is intended to provide early cues forpre-conditioned splice points. This message may provide early cues toassist digital program insertion. Concatenation devices may make use ofthis information to generate an IDR prior pictures output cue message.

N_num_frames_to_prior_pics_dpb_output specifies the number of frames,from decoding of current frame to the decoding of frame immediatelyprior to the indicated splice point to which this message applies.

M_num_frames_(—) dpb_output_delay specifies the number of frames in DPBwith contiguous output times. The first frame has output time equal toN_num_frames_to_prior_pics_dpb_output+2.

num_prior_frames_in_dpb indicates the number of frames remaining in theDPB to be output after the decoding of the associated frame prior to theidentified potential splice point. Not all pictures in the DPB may haveconsecutive output times.

last_frame_before_splice_POC specifies the picture order count of theframe prior to the identified splice point.

splice_pic_initial_buffering_delay specifies the initial buffering delayeffective at the picture in the stream immediately after the identifiedsplice point.

splice_pic_dpb_output_delay specifies the DPB output delay (e.g.,PTS-DTS) effective at the picture in the stream immediately after theidentified splice point.

Splice Point DPB Cue4 Message Syntax

C Descriptor splice_point_DPB_cue4( payloadSize ) {num_prior_pics_in_dpb 5 u(5) for (j = 0; j < num_prior_pics_in_dpb; j++){ consecutive_poc_flag[ i ] 5 u(1) pic_struct[ i ] 5 u(3)  }

Splice Point DPB Cue4 Message Semantics

The splice point DPB cue message is intended to provide DPB cues forpre-conditioned splice points. The message shall contain information ofthe DPB state after completion of the DPB process as described in clauseC.2 of ITU Recommendation H.264 for the associated access unit. Thismessage may provide cues to assist digital program insertion.Concatenation devices may make use of this information to generate IDRprior pictures output cue message.

num_prior_pics_in_dpb indicates the number of prior pictures remainingin the DPB to be output after the decoding of the associated accessunit.

consecutive_poc_flag[i] equal to 1 indicates that the picture hasconsecutive picture order count as compared to previous picture outputfrom the DPB. Picture order counts are consecutive if the increments are1 for field pictures and 2 for frame pictures, or the pictures areintended to be output without gaps. The pictures are sorted bycorresponding picture order counts from smallest value to highest value.

pic_struct[i] indicates the pic_struct value that is associated witheach picture. The pictures are sorted by corresponding picture ordercounts from smallest value to highest value.

Having described some example message syntaxes, it should be appreciatedthat one method embodiment, shown in FIG. 5 and denoted as method 500,comprises providing a video stream in a transport stream, the firstvideo stream comprising a plurality of compressed pictures (502), andproviding information in the video stream, wherein the informationincludes plural data fields, the provided information comprising: afirst data field corresponding to a location in the video stream of apotential splice point in the video stream, wherein the value of thefirst data field identifies a location in the video stream after thelocation of the provided information in the video stream; a second datafield corresponding to decompressed pictures yet to be output (DPYTBO)by a video decoder at the identified potential splice point (IPSP) whenthe video decoder decompresses the video stream, wherein the value ofthe second data field is a number corresponding to the DPYTBO by thevideo decoder at the IPSP; and a third data field corresponding topictures with contiguous output times (WCOT), wherein the value of thethird field corresponds to a set of pictures WCOT of the DPYTBO by thevideo decoder at the IPSP (504).

Another method embodiment, shown in FIG. 6 and denoted as method 600,comprises providing a video stream in a transport stream, the firstvideo stream comprising a plurality of compressed pictures (602), andproviding information in the video stream, wherein the informationincludes plural data fields, the provided information comprising: afirst data field corresponding to decompressed pictures yet to be output(DPYTBO) by a video decoder at a potential splice point (IPSP) when thevideo decoder decompresses the video stream, wherein the value of thefirst data field is a number corresponding to the DPYTBO by the videodecoder at the IPSP; and a plurality of second data fields, wherein eachof the plurality of second data fields corresponds to a respectiveDPYTBO by the video decoder at the IPSP, wherein the number of seconddata fields in the plurality of second data field equals to the value ofthe first data field, and wherein the value of each of the second datafield indicates how each of the respectively corresponding DPYTBO at theIPSP is to be output by the video decoder (604).

Another method embodiment, shown in FIG. 7 and denoted as method 700,comprises providing a video stream in a transport stream, the firstvideo stream comprising a plurality of compressed pictures (702), andproviding information in the video stream, wherein the informationincludes plural data fields, the provided information comprising: afirst data field corresponding to decompressed pictures yet to be output(DPYTBO) by a video decoder at a potential splice point (IPSP) when thevideo decoder decompresses the video stream, wherein the value of thefirst data field is a number corresponding to the DPYTBO by the videodecoder at the IPSP; and a plurality of second data fields that provideinformation corresponding to the relative output times of the DPYTBO bythe video decoder at the IPSP (704).

In addition, one having ordinary skill in the art, in the context of thepresent disclosure, should appreciate that receive-side (e.g., at avideo processing device such as the splicer 112) method embodiments arecontemplated that largely mirror the transmit-side methods describedabove in association with FIGS. 5-7. Accordingly, one method embodiment,shown in FIG. 8 and denoted as method 800, comprises receiving a videostream in a transport stream, the first video stream comprising aplurality of compressed pictures (802), and receiving information in thevideo stream, wherein the information includes plural data fields, thereceived information comprising: a first data field corresponding to alocation in the video stream of a potential splice point in the videostream, wherein the value of the first data field identifies a locationin the video stream after the location of the received information inthe video stream; a second data field corresponding to decompressedpictures yet to be output (DPYTBO) by a video decoder at the identifiedpotential splice point (IPSP) when the video decoder decompresses thevideo stream, wherein the value of the second data field is a numbercorresponding to the DPYTBO by the video decoder at the IPSP; and athird data field corresponding to pictures with contiguous output times(WCOT), wherein the value of the third field corresponds to a set ofpictures WCOT of the DPYTBO by the video decoder at the IPSP (804).

Another method embodiment, shown in FIG. 9 and denoted as method 900,comprises receiving a video stream in a transport stream, the firstvideo stream comprising a plurality of compressed pictures (902), andreceiving information in the video stream, wherein the informationincludes plural data fields, the received information comprising: afirst data field corresponding to decompressed pictures yet to be output(DPYTBO) by a video decoder at a potential splice point (IPSP) when thevideo decoder decompresses the video stream, wherein the value of thefirst data field is a number corresponding to the DPYTBO by the videodecoder at the IPSP; and a plurality of second data fields, wherein eachof the plurality of second data fields corresponds to a respectiveDPYTBO by the video decoder at the IPSP, wherein the number of seconddata fields in the plurality of second data field equals to the value ofthe first data field, and wherein the value of each of the second datafield indicates how each of the respectively corresponding DPYTBO at theIPSP is to be output by the video decoder (904).

Another method embodiment, shown in FIG. 10 and denoted as method 1000,comprises receiving a video stream in a transport stream, the firstvideo stream comprising a plurality of compressed pictures (1002), andreceiving information in the video stream, wherein the informationincludes plural data fields, the received information comprising: afirst data field corresponding to decompressed pictures yet to be output(DPYTBO) by a video decoder at a potential splice point (IPSP) when thevideo decoder decompresses the video stream, wherein the value of thefirst data field is a number corresponding to the DPYTBO by the videodecoder at the IPSP; and a plurality of second data fields that provideinformation corresponding to the relative output times of the DPYTBO bythe video decoder at the IPSP (1004).

In some embodiments, additional (or fewer) steps for one or more of theabove-described methods are contemplated to be within the scope of thedisclosure.

Further, one having ordinary skill in the art, in the context of thepresent disclosure, should understand that any of the above methods maybe considered without including explicitly the consecutive_poc_flag ofeach picture in the DPB and its pic_struct. Likewise, it should beunderstood, in the context of the present disclosure, that any of theabove methods could be amended to include any of the syntax elementsdescribed in other parts of this disclosure or the sections entitled:Forewarning location of end_of_stream in spliced bitstreams, or/and inthe section entitled: Method to control DPB output in non-seamlessspliced bitstreams with end_of_stream.

Method to Control DPB Output in Non-Seamless Spliced Bitstreams withEnd_of_Stream

Having described the conveyance of information between an encoder 111and video processing device (e.g., splicer 112), attention is directedto certain system and/or method embodiments for providing, receiving,and/or processing of control information, the control information (orinformation) passed through by the splicer 112 from the encoder 111 insome embodiments, or generated by the splicer 112 (e.g., in response tothe information received from the encoder 111 as described above) insome embodiments. Such control information enables a video processingdevice, such as the DHCT 200, to orchestrate the output of pictures fromthe DPB 297. The message according to the following description providesinformation to control the output of DPB pictures at the splice point ofnon-seamless concatenated bitstreams. For instance, and referring toFIG. 3, relevant to this discussion are machine states of the DPB 297where M equals, for instance, two (2). The message can serve as a toolto splicing devices along with the end_of_stream NAL unit andno_output_of_prior_pics_flag. The message is provided in the bitstreamprior to the end_of_stream NAL unit to identify its location and specifythe output behavior of non-previously output pictures in the DPB 297 atthe time that the end_of_stream NAL unit becomes effective. Informationthat specifies the output behavior of each non-previously output DPBpicture allows for outputting a picture, not outputting, or outputtingthe picture for a number of consecutive times prior to outputting thesubsequent picture from the first bitstream. The message may be signaledahead with information that points to the location of the end_of_stream

One method provides a message that serves as a cue that is provided by asplicing device 112 to a decoder 222 to convey information thatalleviates the non-seamless transition to a concatenated stream. Asdescribed herein, the concept of N may be used to identify the locationof a splice point in the bitstream and may be expressed in the messagein one of different forms that connotes the number of consecutiveframes, access units, or slices in the bitstream to identify thelocation of the splice point from the location of the message. Differentmethod embodiments for expressing N in the message are disclosed herein.One method embodiment does not convey N but only information to controlthe output of pictures.

A potential splice point may be announced in one or more, and possiblyall of the N access units preceding the location of the splice point.The value of N is decremented in successive instances of the message inconsecutive access units that precede the splice point

A decoder, such as decoder or decompression engine 222, may use theprovided message to alleviate the effects of a non-seamlessconcatenation of two bitstreams. At the decoder 222, the concatenatedbitstream exhibits a transition period that spans M picture-outputtimes. As a picture from the first bitstream is output from the DPB 297during the transition period, control information specifies theoutputting of each picture. Outputting may be consistent withpic_struct, but for an interlaced source, the splicing device 112provides information to prohibit the manifestation of motion jitter.Hence, for an interlaced source, the last output field of an interlacedframe is output, as both the top and bottom fields, to satisfy therepetition amount specified by the output cue in the message.

In similar fashion to the disclosure above, some example syntaxes areprovided below, from which various systems and methods are contemplatedthat convey or receive such information and process the same.

IDR Prior Pictures Output Cue Message Syntax

C Descriptor IDR_prior_pic_output_cue( payloadSize ) {use_no_output_of_prior_pics_flag 5 u(1) num_prior_pics_in_dpb 5 u(5) for(j = 0; j < num_prior_pics_in_dpb; j++) { output_cue[ i ] 5 u(3)  }

IDR Prior Pictures Output Cue Message Semantics

The IDR prior pic output cue message is intended to assist the decoder222 in outputting prior pictures remaining in the DPB 297 from theprevious bitstream across a concatenation. This message applies to theprior pictures output by the DPB process triggered by the decoding of anEnd_of_Stream NAL unit or an IDR access unit. The message shall be inthe same access unit as the effected End_of_Stream NAL unit, or if thereis no End_of_Stream NAL unit, in the same access unit as the effectedIDR NAL. This message may be useful for digital program insertionapplications. Further, it may help the decoder 222 to maintaincontinuous picture output for bitstreams having a discontinuous DPBoutput across concatenation points. In such applications, this messagemay be output by concatenation devices (e.g., splicer 112).

use_no_output_of_prior_pics_flag equal to 1 specifies that the decodershould use the no_output_of_prior_pics_flag as specified in the IDRaccess unit regardless of the inference rules as specified by [sectionx.x], and across End_of_Stream NAL unit.

num_prior_pics_in_dpb indicates the number of prior pictures remainingin the DPB 297 to be output during the decoding of the IDR access unit.

output_cue[i] indicates cues for the output of each prior picture in theDPB 297, according to Table X-X below. The pictures are sorted bycorresponding picture order counts from smallest value to highest value.The decoder 222 may make use of the output_cue for better visualexperience.

TABLE X-X Value Recommended output cue for picture 0 Picture may bediscarded without output 1 Picture may be output according to picture'spic_struct* 2 Picture's output as indicated by pic_struct* may berepeated once 3 Picture's output as indicated by pic_struct* may berepeated twice 4 Picture's output as indicated by pic_struct* may berepeated indefinitely until first picture is output from DPB after thedecoding of the IDR access unit 5 . . . 7 Reserved

Note (denoted by an astericks) that for interlaced video sources,repetition may mean that the last output field of an interlaced frame isoutput as both the top and bottom fields to satisfy the number ofrepetitions specified by output cue. Alternatively, repetition bycausing a field to be displayed for both and top fields may require thedecoder 222 to perform anti-flicker or anti-fluttering filter on theoutput field. Alternatively, an instance of the field is output filtered(e.g., when the field is output as top field) and the second instance isnot (e.g., bottom field). Or, the fields of an interlaced pictures maybe de-interlaced by a de-interlacing method in the decoder 222.

Another example message syntax is as follows:

IDR Prior Pictures Output Cue Message Syntax

C Descriptor IDR_prior_pic_output_cue( payloadSize ) {num_prior_pics_in_dpb 5 u(5) for (j = 0; j < num_prior_pics_in_dpb; j++){ output_cue[ i ] 5 u(3)  }

IDR Prior Pictures Output Cue Message Semantics

The IDR prior pic output cue message is intended to assist the decoder222 in outputting prior pictures remaining in the DPB 297 from theprevious bitstream across a concatenation. This message applies to theprior pictures output by the DPB process triggered by the decoding of anEnd_of_Stream NAL unit or an IDR access unit. The message shall be inthe same access unit as the effected End_of_Stream NAL, or if there isno End_of_Stream NAL, in the same access unit as the effected IDR NAL.This message may be useful for digital program insertion applications.Further, it may help the decoder 222 to maintain continuous pictureoutput for bitstreams having discontinuous DPB output acrossconcatenation points. In such applications, this message may be outputby concatenation devices.

num_prior_pics_in_dpb indicates the number of prior pictures remainingin the DPB 297 to be output during the decoding of the IDR access unit.

output_cue[i] indicates cues for the output of each prior picture in theDPB 297, according to Table X-X1. The pictures are sorted bycorresponding picture order counts from smallest value to highest value.The decoder 222 may make use of the output_cue for better visualexperience.

TABLE X-X1 Value Recommended output cue for picture 0 Picture may bediscarded without output 1 Picture may be output according to picture'spic_struct* 2 Picture's output as indicated by pic_struct* may berepeated once 3 Picture's output as indicated by pic_struct* may berepeated twice 4 Picture's output as indicated by pic_struct* may berepeated indefinitely until first picture is output from DPB after thedecoding of the IDR access unit 5 Repeat picture output as indicated bypic_struct* until the output time of the next picture in the DPB. 6Output the last picture once again* 7 Reserved

Note (denoted by an astericks) that for an interlaced video source,repetition may mean that the last output field of an interlaced frame isoutput as both the top and bottom fields to satisfy the number ofrepetitions specified by output cue. Alternatively, repetition bycausing a field to be displayed for both and top fields may require thedecoder 222 to perform anti-flicker or anti-fluttering filter on theoutput field. Alternatively, an instance of the field is output filtered(e.g., when the field is output as top field) and the second instance isnot (e.g., bottom field). Or, the fields of an interlaced pictures maybe de-interlaced by a de-interlacing method in the decoder 222.

Any of the above example syntax and corresponding methods of thissection may be amended to include any of the syntax elements describedin other parts of this disclosure or in the sections entitled:Forewarning location of end_of_stream in spliced bitstreams, or/and inConveyance of suitable splice points in the bitstream.

Forewarning Location of End_of_Stream in Spliced Bitstreams

One method described below provides for transmitting or receiving amessage that conveys the location of an end_of_stream NAL unit in thebitstream. The end_of_stream NAL unit is the last NAL unit in the accessunit that ends a bitstream. In some system and/or method embodiments, anew bitstream may immediately follow that access unit that ended thebitstream. The provided message conveys an end of stream in thebitstream located N access units from the location of the message in thebitstream. The provided or received message may serve as a cue that canbe provided by a splicing device 112 or other equipment to convey to adecoder, such as decoder 222, the location of the forthcomingend_of_stream NAL unit. The end_of_stream is located N access units fromthe current access unit or the location of the message.

As some receivers may potentially process and interpret non-VCL (nonvideo coding layer) NAL units in the back-end, a decoder may use theinformation in the provided or received message to anticipate and plan atransition from a first to a second bitstream.

As N identifies the location of an end_of_stream NAL unit, it may beexpressed in the message in one of different forms that connotes thenumber of consecutive frames, access units, or slices in the bitstreamto identify the location of end_of_stream from the location of theprovided or received message. Several syntax (and hence methods) ofexpressing N in the message are described below.

End_of_Stream Cue1 Message Syntax

C Descriptor end_of_Stream_cue1( payloadSize ) {N_num_fields_to_end_of_stream 5 u(8)  }

End_of_Stream Cue1 Message Semantics

The end_of_stream cue1 message is intended to provide forewarning of theend_of_stream to allow ample time for decoders, such as decoder 222, toanticipate and plan a change from a first to a second bitstream.

N_num_fields_to_end_of_stream specifies the time, in field output timeunits, from decoding of current access unit to the decoding of accessunit immediately prior to the end_of_stream to which this messageapplies.

End_of_Stream Cue2 Message Syntax

C Descriptor end_of_Stream _cue2( payloadSize ) {N_num_AU_to_end_of_stream 5 u(8)  }

End_of_Stream Cue2 Message Semantics

The end_of_stream cue2 message is intended to provide forewarning of theend_of_stream to allow ample time for decoders to anticipate and plan achange from a first to a second bitstream.

N_num_AU_to_end_of_stream specifies the number of access units, fromdecoding of current access unit to the decoding of access unitimmediately prior to the end_of_stream to which this message applies.

End_of_Stream Cue3 Message Syntax

C Descriptor splice_point_cue3( payloadSize ) {N_num_frames_to_end_of_stream u(8) }

End_of_Stream Cue3 Message Semantics

The end_of_stream cue3 message is intended to provide forewarning of theend_of_stream to allow ample time for decoders to anticipate and plan achange from a first to a second bitstream.

N_num_frame_to_end_of_stream specifies the number of frames, fromdecoding of current frame to the decoding of frame immediately prior tothe end_of_stream to which this message applies.

Any of the above syntaxes (and hence corresponding methods) may beamended to include any of the syntax elements described in other partsof this document or in the section entitled: Method to control DPBoutput in non-seamless spliced bitstreams with end_of_stream, or/and inthe section entitled: Conveyance of suitable splice points in thebitstream.

Any process descriptions or blocks in flow diagrams should be understoodas representing modules, segments, or portions of code which include oneor more executable instructions for implementing specific logicalfunctions or steps in the process, and alternate implementations areincluded within the scope of the embodiments in which functions may beexecuted out of order from that shown or discussed, includingsubstantially concurrently or in reverse order, depending on thefunctionality involved, as would be understood by those reasonablyskilled in the art of the present disclosure.

Additionally, though the various methods are described above in thecontext of the implementation environment disclosed in FIGS. 1 and 2, itshould be appreciated that other implementation environments arecontemplated to be within the scope of the disclosure, and hence thevarious methods described herein are not limited to the environment inthose figures.

It should be emphasized that the above-described embodiments of thepresent disclosure are merely possible examples of implementations,merely set forth for a clear understanding of the principles of thedisclosure. Many variations and modifications may be made to theabove-described embodiment(s), and all such modifications and variationsare intended to be included herein within the scope of this disclosureand protected by the following claims.

At least the following is claimed:
 1. A method, comprising: receiving avideo stream in a transport stream, the first video stream comprising aplurality of compressed pictures; and receiving information in the videostream, wherein the information includes plural data fields, thereceived information comprising: a first data field corresponding todecompressed pictures yet to be output (DPYTBO) by a video decoder at apotential splice point (IPSP) when the video decoder decompresses thevideo stream, wherein the value of the first data field is a numbercorresponding to the DPYTBO by the video decoder at the IPSP; and aplurality of second data fields, wherein each of the plurality of seconddata fields corresponds to a respective DPYTBO by the video decoder atthe IPSP, wherein the number of second data fields in the plurality ofsecond data field equals to the value of the first data field, andwherein the value of each of the second data field indicates how each ofthe respectively corresponding DPYTBO at the IPSP is to be output by thevideo decoder.
 2. The method of claim 1, wherein the value of one of theplurality of the second data fields corresponds to foregoing the outputof the picture corresponding to the respective second data field.
 3. Themethod of claim 1, wherein the value of one of the plurality of thesecond data fields corresponds to allowing the output of the picturecorresponding to the respective second data field to be repeated once.4. The method of claim 2, wherein the value of another of the pluralityof the second data fields corresponds to allowing the output of thepicture corresponding to the respective second data field to be repeatedtwice.
 5. The method of claim 1, wherein the value of one of theplurality of the second data fields corresponds to allowing the outputof the picture corresponding to the respective second data field to berepeated one of more than once or until the output of next picture to beoutput from the DPYTBO.
 6. The method of claim 1, wherein the receivedinformation in the video stream is provided in a message in the videostream.
 7. The method of claim 1, wherein the received informationfurther comprises of a third data field wherein the value of the thirddata field represents a number of pictures.
 8. The method of claim 1,wherein the received information further comprises of a fourth datafield, wherein the fourth data field identifies the last picture in aset of pictures with contiguous output times of the DPYTBO.
 9. Themethod of claim 1, wherein the received information further comprises ofa fifth data field, wherein the fifth data field provides an initialdelay of a compressed picture buffer at the IPSP when the video streamis decompressed.
 10. The method of claim 1, wherein the receivedinformation further comprises of an sixth data field, wherein the sixthdata field provides an output picture delay of a decoded picture bufferat the IPSP when the video stream is decompressed.
 11. The method ofclaim 1, wherein the received information further comprises of a seventhdata field, wherein the seventh data field provides an output picturedelay of a decoded picture buffer at the IPSP when the video stream isdecompressed.
 12. A method, comprising: providing a video stream in atransport stream, the first video stream comprising a plurality ofcompressed pictures; and providing information in the video stream,wherein the information includes plural data fields, the providedinformation comprising: a first data field corresponding to decompressedpictures yet to be output (DPYTBO) by a video decoder at a potentialsplice point (IPSP) when the video decoder decompresses the videostream, wherein the value of the first data field is a numbercorresponding to the DPYTBO by the video decoder at the IPSP; and aplurality of second data fields, wherein each of the plurality of seconddata fields corresponds to a respective DPYTBO by the video decoder atthe IPSP, wherein the number of second data fields in the plurality ofsecond data field equals to the value of the first data field, andwherein the value of each of the second data field indicates how each ofthe respectively corresponding DPYTBO at the IPSP is to be output by thevideo decoder.
 13. The method of claim 12, wherein the value of one ofthe plurality of the second data fields corresponds to foregoing theoutput of the picture corresponding to the respective second data field.14. The method of claim 12, wherein the value of one of the plurality ofthe second data fields corresponds to allowing the output of the picturecorresponding to the respective second data field to be repeated once.15. The method of claim 13, wherein the value of another of theplurality of the second data fields corresponds to allowing the outputof the picture corresponding to the respective second data field to berepeated twice.
 16. The method of claim 12, wherein the value of one ofthe plurality of the second data fields corresponds to allowing theoutput of the picture corresponding to the respective second data fieldto be repeated one of more than once or until the output of next pictureto be output from the DPYTBO.
 17. The method of claim 12, wherein theprovided information in the video stream is provided in a message in thevideo stream.
 18. A method, comprising: providing a video stream in atransport stream, the first video stream comprising a plurality ofcompressed pictures; and providing information in the video stream,wherein the information includes plural data fields, the providedinformation comprising: a first data field corresponding to decompressedpictures yet to be output (DPYTBO) by a video decoder at a potentialsplice point (IPSP) when the video decoder decompresses the videostream, wherein the value of the first data field is a numbercorresponding to the DPYTBO by the video decoder at the IPSP; and aplurality of second data fields that provide information correspondingto the relative output times of the DPYTBO by the video decoder at theIPSP.
 19. The method of claim 18, wherein the provided informationconveys repeating the output of at least one of the DPYTBO by the videodecoder at the IPSP.
 20. The method of claim 18, wherein the providedinformation conveys a gap in output of the DPYTBO by the video decoderat the IPSP.